The Use of Transition Entropy in Partially Observable Markov Decision Processes1

In this report we describe a new POMDP algorithm, denoted TEQ-MDP, which computes the optimal policy of a modified MDP and uses the obtained optimal solution to compute the action for the POMDP, as a function of the belief-state. The modified MDP includes state entropy information (transition entropy) in its reward structure so as to value actions that gather information. This algorithm is suitable for real-time implementation, since the main computational burden can be done off-line. We present the results from several tests in example-environments from the literature and compare the performance of the TEQ-MDP algorithm with the performance of another heuristic method (Q-MDP). In the various situations where Q-MDP presents near-optimal behaviour, the TEQ-MDP algorithm performs no worse. Furthermore, TEQ-MDP also presents near-optimal performance in particular cases where the Q-MDP algorithm clearly fails.